Papers by Luca Di Liello
Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for answer sentence selection (AS2) are not yet available for AS2 . |
| Approach: | They propose to incorporate paragraph-level semantics within and across documents to improve transformers for AS2 . they propose to use a dataset to predict whether two sentences are extracted from the same paragraph . |
| Outcome: | The proposed model outperforms baseline models on public and industrial datasets on three public and one industrial dataset. |
Effective Pretraining Objectives for Transformer-based Autoencoders (2022.findings-emnlp)
Copied to clipboard
| Challenge: | ELECTRA is more accurate than BERT, but it is not clear if this is due to its innovative architecture or to the long and extensive training, which highly increases the computation cost for obtaining the final language model. |
| Approach: | They propose to replace BERT’s Masked Language Modeling objective (MLM) with Token Detection (TD) by using a statistical approach to generate light tokens. |
| Outcome: | The proposed method can replace ELECTRA's computationally heavy generators without a significant drop in performance. |
Context-Aware Transformer Pre-Training for Answer Sentence Selection (2023.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to perform Answer Sentence Selection (AS2) using only the candidate sentence are sub-optimal. |
| Approach: | They propose to use pre-trained transformers to perform contextual AS2 fine-tuning . they propose to apply pre-training objectives to local contextual AS2. |
| Outcome: | The proposed methods improve baseline AS2 accuracy by up to 8% on some datasets. |
Paragraph-based Transformer Pre-training for Multi-Sentence Inference (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained transformers perform poorly for multi-candidate inference tasks. |
| Approach: | They propose a pre-training objective that models paragraph-level semantics across multiple input sentences. |
| Outcome: | The proposed model outperforms existing models on three AS2 and one fact verification datasets. |